Measuring the behavioral impact of machine translation quality improvements with A/B testing

In this paper we discuss a process for quantifying the behavioral impact of a domaincustomized machine translation system deployed on a large-scale e-commerce platform. We discuss several machine translation systems that we trained using aligned text from product listing descriptions written in multiple languages. We document the quality improvements of these systems as measured through automated quality measures and crowdsourced human quality assessments. We then measure the effect of these quality improvements on user behavior using an automated A/B testing framework. Through testing we observed an increase in key ecommerce metrics, including a significant increase in purchases.